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Why GAO Did This Study 

Amid efforts to improve performance 
and constrain spending, federal 
agencies are being asked to expand 
the use of rigorous program 
evaluation in decision-making. In 
addition to performance data, 
indepth program evaluation studies 
are often needed for assessing 
program impact or designing 
improvements. Agencies can also use 
their evaluation resources to provide 
information needed for effective 
management and legislative 
oversight. 

GAO was asked to study federal 
agencies with mature evaluation 
capacity to examine (1) the criteria, 
policies, and procedures they use to 
determine programs to review, and 
(2) the influences on their choices. 
GAO reviewed agency materials and 
interviewed officials on evaluation 
planning in four agencies in three 
departments with extensive 
evaluation experience: Education, 
Health and Human Services (HHS), 
and Housing and Urban Development 
(HUD). 

HHS and HUD agreed with the 
description of how they plan 
evaluations. HHS noted that the 
optimal location of evaluation units 
will vary with the circumstances and 
purpose of evaluations. HUD felt the 
draft report did not emphasize 
enough the influence of the 
appropriations process. GAO has 
added text to note its influence on 
evaluation planning. Education 
provided technical comments. 

What GAO Recommends 

GAO makes no recommendations. 

View GAO-1 1-176 or key components. 

For more information, contact Nancy 
Kingsbury, Managing Director, (202) 512-2700 
or kingsburyn@gao.gov 



PROGRAM EVALUATION 

Experienced Agencies Follow a Similar Model for 
Prioritizing Research 



What GAO Found 

Although no agency GAO reviewed had a formal policy describing evaluation 
planning, all followed a generally similar model for developing and selecting 
evaluation proposals. Agencies usually planned an evaluation agenda over 
several months in the context of preparing spending plans for the coming 
fiscal year. Evaluation staff typically began by consulting with a variety of 
stakeholders to identify policy priorities and program concerns. Then with 
program office staff they identified the key questions and concerns and 
developed initial proposals. Generally, the agencies reviewed and selected 
proposals in two steps: develop ideas to obtain initial feedback from senior 
officials and develop full-scale evaluation proposals for review and approval. 

The four general criteria these mature agencies use to plan evaluations were 
remarkably similar: (1) strategic priorities representing major program or 
policy area concerns or new initiatives, (2) program-level problems or 
opportunities, (3) critical unanswered questions or evidence gaps, and (4) the 
feasibility of conducting a valid study. 

The agencies’ procedures differed on some points. External parties’ 
participation in evaluation planning may reflect these agencies’ common 
reliance on nonfederal program partners. Only the offices GAO reviewed in 
HHS’ Centers for Disease Control and Prevention held formal competitions to 
rank-order proposals before submitting them for approval; in other agencies, 
senior officials assessed proposals in a series of discussions. When evaluation 
authority and funds are tied to the program, evaluators generally choose not 
which programs to evaluate but which research questions to answer. 
Sometimes this resulted in a program’s never being evaluated. 

Evaluation units at higher organizational levels conducted a wider range of 
analytic activities, consulted more formally with program offices, and had less 
control over approvals. The Congress influences an agency’s program 
evaluation choices through legislating evaluation authority, mandating 
studies, making appropriations, and conducting oversight. 

GAO concludes that 

• all four criteria appear key to setting an effective evaluation agenda that 
provides credible, timely answers to important questions; 

• most agencies could probably apply the general model in which 
professional evaluators iteratively identify key questions in consultation 
with stakeholders and then scrutinize and vet research proposals; 

• agencies could adapt the model and decide where to locate evaluation 
units to meet their own organizational and financial circumstances and 
authorities; and 

• agencies’ reaching out to key program and congressional stakeholders in 
advance of developing proposals could help ensure that their evaluations 
will be used effectively in management and legislative oversight. 
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Abbreviations 



ACF Administration for Children and Families 

ASPE Office of the Assistant Secretary for Planning and 

Evaluation 

CDC Centers for Disease Control and Prevention 

DASH Division of Adolescent and School Health 

DHAP Division of HIV/AIDS Prevention 

DNPAO Division of Nutrition, Physical Activity, and Obesity 
GPRA Government Performance and Results Act 

HHS Department of Health and Human Services 

HUD Department of Housing and Urban Development 

IES Institute of Education Sciences 

NCEE National Center for Education Evaluation and Regional 

Assistance 

NRC National Research Council 

OMB Office of Management and Budget 

OPEPD Office of Planning, Evaluation, and Policy Development 

OPRE Office of Planning, Research and Evaluation 

PART Program Assessment Rating Tool 

PD&R Office of Policy Development and Research 

PHS Public Health Service 

PPSS Policy and Program Studies Service 

TANF Temporary Assistance to Needy Families 
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United States Government Accountability Office 
Washington, DC 20548 



January 14, 2011 

The Honorable Daniel K. Akaka 
United States Senate 

Dear Senator Akaka: 

Amid efforts to improve federal government performance and constrain 
federal spending, the Obama Administration has emphasized expanding 
the availability and use of rigorous program evaluation. The aim is to 
establish an evaluation culture in which agencies regularly use evidence to 
inform program design and investment decisions, complementing existing 
efforts to strengthen performance measurement and management. Today, 
most federal agencies use performance measures to track progress toward 
goals. Few appear to conduct in-depth program evaluations regularly to 
assess their programs’ impart or inform policymakers on how to improve 
results. Increasing demands for performance informat ion prompted your 
expressed interest in how agencies, given their limited evaluation 
resources, can provide the information that is necessary for effective 
management and legislative oversight. To help agencies plan their 
evaluation activities strategically, you asked us to study federal agencies 
that have mature program evaluation capacity, to learn 

1. What criteria, policies, and procedures do they use to determine which 
programs to review? 

2. What conditions influence an agency’s choices? 

To find answers that would apply broadly to other federal agencies about 
how to prioritize program evaluations, we selected four agencies in three 
departments with known evaluation capacity — the ability to systematically 
collect, analyze, and use data on program results — in a variety of 
substantive areas. The four agencies were the Department of Education 
and the Department of Housing and Urban Development (HUD) and two 
agencies in the Department of Health and Human Services (HHS) — the 
Administration for Children and Families (ACF) and the Centers for 
Disease Control and Prevention (CDC). 

To identify agencies fulfilling our criteria, we reviewed previous GAO 
reports and agency documents for evidence of emphasis on conducting 
evaluations. For example, we searched for examples of agencies’ 
incorporating the results of program evaluations in their annual 
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performance reports. To obtain a diverse set of cases, we selected 
agencies that addressed a variety of content areas and program types — 
direct services or grants, regulation, and research — and that varied in the 
conditions likely to affect their decision process, such as whether they had 
a central office responsible for evaluation. To identify evaluation planning 
procedures and criteria, we reviewed agency materials and interviewed 
agency evaluation officials about their evaluation planning process. We 
identified conditions influencing their choices in interviews with agency 
officials and in comparisons of what we found across the agencies. 

We conducted this performance audit from May 2010 through December 
2010 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonable basis for 
our findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our findings 
and conclusions based on our audit objectives. We requested comments 
on a draft of this report from the Secretaries of Education, Health and 
Human Services, and Housing and Urban Development. HHS and HUD 
provided comments that are reprinted in appendixes I and II and described 
at the end of this letter. Education provided technical comments. 



Background 



Under the Government Performance and Results Act of 1993 (GPRA) 
federal agencies are expected to focus on achieving results and to 
demonstrate, in annual performance reports and budget requests, how 
their activities help achieve agency or governmentwide goals. In 2002, to 
encourage greater use of program performance information in decision 
making, the Office of Management and Budget (OMB) created the Program 
Assessment Rating Tool (PART). PART was intended to provide a 
consistent approach for evaluating federal programs within the executive 
budget formulation process. However, because PART conclusions rely on 
available program performance and evaluation information, many of the 
initial recommendations focused on improving outcome and efficiency 
measures. Although GPRA and PART helped improve the availability of 
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better performance measures, we and OMB have noted that this did not 
result in their greater use by the Congress or agencies. 1 

In October 2009, OMB announced a plan to strengthen federal program 
evaluation, noting that rigorous independent program evaluations can help 
determine whether government programs are achieving their intended 
outcomes as well as possible and at the lowest possible cost. 2 Program 
evaluations are systematic studies that assess how well a program is 
working, and they are individually tailored to address the client’s research 
question. 

• Process (or implementation) evaluations assess the extent to which a 
program is operating as intended. 

• Outcome evaluations assess the extent to which a program is achieving its 
outcome-oriented objectives but may also examine program processes to 
understand how outcomes are produced. 

When external factors such as economic or environmental conditions are 
known to influence a program’s outcomes, 

• Impact evaluations may be used to measure a program’s net effect by 
comparing outcomes with an estimate of what would have occurred had 
there been no program intervention. 

Thus, program evaluation can provide an important complement to agency 
performance data that simply track progress toward goals. 

In announcing the evaluation initiative, the OMB Director expressed 
concern that many important programs had never been evaluated, 
evaluations had not sufficiently shaped budget priorities or management 
practices, and many agencies lack an evaluation office capable of 
supporting an ambitious strategic research agenda. The initiative consisted 



‘GAO, Managing for Results: Enhancing Agency Use of Performance Information for 
Management. Decision Making,. GAO-05-927 (Washington, D.C.: Sept. 9, 2005); Executive 
Office of the President, Office of Management and Budget, Analytical Perspectives — 
Budget of the United States Government, Fiscal Year 2011 (Washington, D.C.: The White 
House, Feb. 1, 2010), pp. 73-74. 

2 Office of Management and Budget, Increased Emphasis on Program Evaluations, M-10- 
01, Memorandum for the Heads of Executive Departments and Agencies (Washington, D.C.: 
The White House, Oct. 7, 2009). 



Page 3 



GAO-11-176 Prioritizing Evaluation 



